Repository navigation
Conversation
CometFairMemoryPool limited each consumer that can spill to (task_share - unspillable) / num_spill, where num_spill counted every registered consumer that can spill. DataFusion registers each operator's consumer when it builds the stream tree, before any batch flows, so the operators above a blocking one counted while they held nothing. A sort below a window and four aggregates was held to a sixth of the task's share and spilled while most of it was unused. Count only the active consumers that can spill, plus the one asking. A consumer is active while it holds more than its low-water mark, the least it has held since its first reservation, because from DataFusion 56 an aggregate reserves its initial table when the plan starts (apache/datafusion#25900). The task's total is still checked against the task's share, so a consumer that starts later is refused until the others give memory back. A sort below an OrderedFinalAggregateStream, which apache#7096 held to half of the share, can now fill the share while the aggregate is idle. It gives part of that back before the aggregate's first request, which a test checks with the share set to the sort's own peak. The apache#6880 test of a PartialMerge over a sort-merge join used a 16 MiB pool because the sorts below the join got a fraction of it. They now get the whole share while they read, so it uses 2 and 8 MiB, like its sibling test. Closes apache#7628.
a467c2c to
ff73d73
Compare
andygrove
left a comment
There was a problem hiding this comment.
Thanks for this, @comphead. The tests that run real DataFusion operators on CometFairMemoryPool make the rule easy to trust, and counting only the consumers doing work is a good fix for #7628. I also ran two SortExecs under a SortMergeJoinExec on the pool, with the share between 100% and 200% of one sort's peak. Every run completed and the total spill count was the same or lower than before the PR, so I don't see the late-starting sort getting starved in that shape.
I have a few questions I'd like answered before this merges.
Benchmark. The description says the A/B on the job from #7628 is still to come. Since this changes when every query spills, could you post it before merge? A run at a tight pool where the operator spills and then reads its spill back would help too. The effect on whole queries (fewer spills where one operator works, later consumers refused until others give memory back) is the part the unit tests can't show.
Ordered final aggregate. #7096 left room for an OrderedFinalAggregateStream by construction, and a refused request there fails the query because it has no spill path. Now the room depends on ExternalSorter giving memory back before the aggregate's first request. a_sort_that_fills_the_share_leaves_room_for_the_final_aggregate_above_it uses with_sort_spill_reservation_bytes(0) and a share equal to the sort's own peak. What happens with the merge reservation Comet actually runs with, which cannot spill and takes its bytes out of the share? Could the test also use an aggregate with many more groups per batch?
DataFusion 55 failure state. The description says an operator below an aggregate can now reach the "Decreasing allocation after spilling should succeed" state on DataFusion 55, and that in the shapes you measured the lower operator gave back far more first. The sort case is pinned by a test. Could the lower aggregate of the count(DISTINCT) chain be pinned at a small share too? Then a DataFusion bump that changes how much it frees fails a test instead of a query.
Single-reservation consumers. low_water is set by the first reservation, so a spillable consumer that reserves once and holds that memory never becomes active and takes no share. Only the task-total check bounds it. Is there a consumer in Comet that behaves this way, such as the shuffle writer or the Iceberg fanout writer? If so, is the task-total check enough for it?
Migration guide. The paragraph you edit is under "Upgrading to Comet 1.2.0" and says "Comet 1.2.0 divides spark.memory.offHeap.size among the tasks running native plans". The description says the per-task shares are not in 1.2.0, and on branch-1.2 I see pool_size / registered consumers with no task division. Should that paragraph move under 1.3.0 together with this change?
… fair pool Restore the count(DISTINCT) chain test, so a DataFusion bump that changes how much the lower aggregate frees fails a test instead of a query. Run the ordered final aggregate test with DataFusion's default merge reservation and with 1024 groups per batch, and check that the task's total keeps the bytes of a consumer that reserved only once. Say that the sort frees its merge reservation before emitting, and move the memory pool note to the 1.3.0 migration section, since apache#7096 is not in 1.2.0.
|
Thanks for the careful review. Each point below is checked against the code or a run. Benchmark. A local A/B of the two shapes from the issue, base against this PR. The native libraries are optimized, the pool is tight, and there are three rounds with alternating order in fresh JVMs:
The second row shows the cost you describe: the shuffle writer starts once the aggregate emits and spills a little more, while the total falls. For spills read back at a tight pool, Ordered final aggregate. I measured the share in use at the aggregate's first request, with the share set to the sort's own peak:
The merge reservation makes room rather than taking it. DataFusion 55 failure state. I restored Single-reservation consumers. Neither Comet consumer that can spill reserves only once. Migration guide. Moved under "Upgrading to Comet 1.3.0". It now says "Comet 1.3.0 divides". |
andygrove
left a comment
There was a problem hiding this comment.
Thanks for working through all of these, @comphead. The measurements on the ordered final aggregate with the real merge reservation answer what I was worried about, and pinning the count(DISTINCT) chain means a DataFusion bump will tell us if that changes. The single-reservation answer makes sense too, since both of Comet's own spillable consumers reserve on every batch.
The fork CI run on 44cda511 failed in the Spark 4.1 exec job, in CometTaskMetricsSuite's "native shuffle task metrics include existing child sort spill metrics once" (CometTaskMetricsSuite.scala:823, "Native child sort did not spill"). I think this is the same effect as the #6880 test you already adjusted. The test sizes the pool so the sort under the native shuffle spills, and the idle shuffle writer no longer takes half the share, so the sort now fits. Could you tighten that test's pool, or shrink its batches, so the sort still spills? Then could you run the whole exec job, since this suite wasn't in the local list?
After that, the cluster A/B on the job from #7628 is the last thing I'd like to see before this merges.
Which issue does this PR close?
Closes #7628.
Rationale for this change
CometFairMemoryPoollimits a consumer that can spill to(task_share - unspillable) / num_spill, andnum_spillcounted every registered consumer that can spill. DataFusion registers each operator's consumer when it builds the stream tree, before any batch flows (SortExec::executeand every aggregate stream's constructor in 55.2). So the operators above a blocking one count while they hold nothing, and the operator doing the work spills while most of the task's share is unused. The issue has the production numbers.A new test reproduces it on
mainwith real DataFusion operators. It sorts the same input in the same pool, sized so the sort fits the task's share:mainSortExecSingleaggregateSpark and Polars avoid the problem with reclaim rather than shares. Spark divides its pool among tasks that hold memory and, within a task, spills other consumers on demand. Polars' out-of-core engine spills registered frames when a global budget is exceeded. Comet cannot ask a native operator to give memory back (#3873, apache/datafusion#21422), so it keeps the shares and counts only the consumers doing work.
What changes are included in this PR?
num_spillnow counts the active consumers that can spill, plus the one asking. A consumer is active while it holds more than its low-water mark, the least it has held since its first reservation. That leaves out a registered consumer that holds nothing, and from DataFusion 56 also an aggregate that holds only the initial table it reserves when the plan starts (fix: reserve initial table memory in aggregate streams datafusion#25900). A consumer that emptied, such as an aggregate that emitted early, is active again once it holds memory, and one that only gives memory back, such as a sort emitting what it buffered, takes no share. The check of the task's total against the task's share is unchanged, so a consumer that starts later is refused until the others give memory back.OrderedFinalAggregateStream, which has no spill path over fully sorted input, was held to half of the share so the aggregate kept the other half. The sort can now fill the share while the aggregate is idle. It gives part of that back before the aggregate's first request: everything if it spilled, and otherwise whatExternalSorterreserves beyond its sorted runs. With the share set to the sort's own peak, the aggregate's first request found 47% of the share in use when the sort emits from one batch and 75% when it merges runs. A test pins this.CometAggregateSuite's "PartialMerge over a sort-merge join on a prefix of its keys spills (issue Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 #6880)" used a 16 MiB pool because the sorts under the join got a fraction of it. They now get the whole share while they read, so 16 MiB no longer makes thePartialMergespill, which failed the fork CI run. It now uses 2 and 8 MiB, like its sibling test, and passes at 2, 4 and 8 MiB.1.2.0orbranch-1.2.holdhelper.DataFusion 55's hash aggregates can fail with "Decreasing allocation after spilling should succeed" when their first request is refused and the task has less free than their emptied table reports: about 8 KiB with a string grouping key, 24 bytes for
collect_list, nothing forcountorhll_sketch_aggwith anInt64key.mainbehaves the same in that state. DataFusion 56 fixes it: apache/datafusion#25900 reserves the initial table up front, so the resize after a spill shrinks, and apache/datafusion#24857 releases the emptied string map. A DataFusion-only test on DataFusionmainconfirmed it. With this PR on DataFusion 55, an operator below that can spill and filled the share while the aggregate was idle can also produce that state, but only if it still holds all but those last bytes when the aggregate starts. In the shapes I measured it gave back far more first: sorts at least a quarter of their peak, and the lower aggregate of acount(DISTINCT)chain at least 3% of the share even when forced. A test pins the chain, so a DataFusion bump that changes how much the lower aggregate frees fails a test instead of a query.#7096's per-task shares are not in
1.2.0orbranch-1.2, so this needs no backport.How are these changes tested?
New tests in
fair_pool.rs, each failing against the pool logic onmain:the_operators_above_a_sort_take_no_share_until_they_are_activeruns real DataFusion plans onCometFairMemoryPoolwith the fake Spark. It is the table above.a_sort_that_fills_the_share_leaves_room_for_the_final_aggregate_above_itsets the share to the sort's own peak and runs anOrderedFinalAggregateStreamabove it, through both the single-batch and the merge emission paths, without and with DataFusion's default 10 MiB merge reservation, and with 256 or 1024 groups in each batch. The sort does not spill and the aggregate completes. Onmainthe sort spills twice.a_final_aggregate_that_starts_while_the_one_below_fills_the_share_spillsruns Spark'scount(DISTINCT)chain (Singlekeyed by(k, d), thenPartialandFinalkeyed byk) with the share at the lower aggregate's own peak and 1 KiB values. TheFinalaggregate starts with 97% of the share in use, spills twice and completes, and the lower aggregate does not spill. Onmainthe lower aggregate spills 6 times.a_consumer_takes_a_share_while_it_holds_more_than_its_low_water_markpins the rule: an initial table takes no share but the task's total keeps its bytes, growth past it takes a share, and an aggregate that emptied takes a share again as soon as it holds memory.I ran the memory pool tests, the Iceberg fanout writer share test and the aggregate planner tests, plus
cargo clippy --all-targets --workspace -- -D warnings,cargo fmtandprettier.Locally on
-Pspark-3.5, because the default 4.1 profile does not resolve offline here, these JVM suites pass:CometAggregateSuite,CometExecSuite,CometExecIteratorLifecycleSuite,CometWindowExecSuite,CometJoinSuite,CometTaskMemoryManagerSuite,CometNativeSuite,CometPluginsSuite,CometRuntimeShutdownSuite,CometInMemoryCacheSuite,CometGenerateExecSuite,CometTopKSuite,CometMergeRowsSuiteandCometExec3_4PlusSuite. That is 769 tests, with 9 canceled by Spark version gates.A local A/B of the two shapes from the issue, base
f043d64367against this PR. The JVM build is the same, the native libraries are optimized (--profile ci), and the pool is tight. There are three rounds with alternating order, each in a fresh JVM, and five runs per query with the first dropped as warm-up. Spills are identical in every round:count(DISTINCT)The first query shows no consistent time difference at this scale. In the second, the aggregate now takes the idle shuffle writer's half. The writer spills a little more, because it starts once the aggregate emits and is refused until the aggregate gives memory back. A final aggregate that spills and reads its spill files back at a tight pool is also covered by
CometAggregateSuite's #6254 tests, which pass with this change.Not run yet: the remaining
execsuites, the Spark SQL tests, and the A/B on the job from the issue, which needs a cluster run.